Method for connecting mobile terminal and external display and apparatus implementing the same

ABSTRACT

A method and apparatus for controlling desktop virtualization function, per-display device window layout provision function, and user input-based external display device utilization function are provided.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of application Ser. No.16/732,839, filed on Jan. 2, 2020; which is a continuation ofapplication Ser. No. 14/022,760, filed on Sep. 10, 2013; which claimspriority under 35 U.S.C. § 119(e) of a U.S. Provisional applicationfiled on Sep. 10, 2012 in the U.S. Patent and Trademark Office andassigned Ser. No. 61/698,909, and a U.S. Provisional application filedon Sep. 10, 2012 in the U.S. Patent and Trademark Office and assignedSer. No. 61/698,985; and which claims priority under 35 U.S.C. § 119(a)of a Korean patent application filed on Sep. 20, 2012 in the KoreanIntellectual Property Office and assigned Serial No. 10-2012-0104794, aKorean patent application filed on Sep. 20, 2012 in the KoreanIntellectual Property Office and assigned Serial No. 10-2012-0104823,and a Korean patent application filed on Jan. 11, 2013 in the KoreanIntellectual Property Office and assigned Serial No. 10-2013-0003465,the entire disclosures of each of which are hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure relates to a function execution method andapparatus of a mobile terminal. More particularly, the presentdisclosure relates to a method and apparatus for managing a connectionbetween a mobile terminal and an external display apparatus.

BACKGROUND

The present disclosure is applicable to Orthogonal Frequency DivisionMultiple Access (OFDMA) system such as 3^(rd) Generation PartnershipProject Long Term Evolution (3GPP LTE) and other similar systems.

With the advance of digital technologies, various types of mobileterminals capable of communicating and processing data (e.g., cellularcommunication terminal, Personal Digital Assistant (PDA), electronicorganizer, smartphone, tablet Personal Computer (PC), and the like) areemerging. Recently, the mobile terminals are evolving intomultifunctional devices integrating various functions in line with themobile convergence tendency. For example, a recent mobile terminalintegrates various functions including voice and video telephonyfunction, a messaging function including Short Message Service (SMS)Multimedia Message Service (MMS), and email, a navigation function, adocument editing (e.g., memo and word processor) function, a picturecapture function, a broadcast playback function, a multimedia (e.g.,video and audio) playback function, an Internet access function, amessenger function, a Social Networking Service (SNS) function, and/orthe like.

The recent mobile terminal supports the external output function capableof connecting to an external display device (e.g., Liquid CrystalDisplay (LCD) monitor) to display the data of the mobile terminalthereon. If the external display device is connected to the mobileterminal, the mobile terminal transfers the terminal's screen to theexternal display device (e.g., when the mobile terminal operates inclone mode). In addition, the mobile terminal may transfer only theexecution screen (e.g., video data according to the motion pictureplayback) of a certain function (or application) which is currentlyrunning on the mobile terminal (e.g., when the mobile terminal operatesin video only mode).

However, the aforementioned methods have a drawback in that when anexternal display device is connected to the mobile terminal, efficientlyutilizing the screens different in size is difficult. For example, themethods according to the related art are limited in function to theclone mode and video only mode.

In addition, the methods according to the related art support only onelayout related to the screen displayed by the mobile terminal such thatthe external display device displays the screen in the same layout asthe mobile terminal. Accordingly, the method according to the relatedart merely enlarges the same layout to fit for the screen size of theexternal display device. As described above, the method according to therelated art has a drawback in that when the mobile terminal is connectedto an external display device, efficiently using the displays differentin size is difficult.

In addition, the dual display system according to the related art has adrawback in cost due to the requirement of extra external input device.Of course, the external input device may be replaced by the input unitof the mobile terminal. However, in the case that the input unit of themobile terminal is a touchscreen, using the touchscreen for controllingthe external display device is difficult.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide a desktop virtualization method and apparatusof a mobile terminal that is capable of operation in the desktopenvironment through the mobile terminal interoperating with an externaldisplay device.

An aspect of the present disclosure is to provide a desktopvirtualization method and apparatus of a mobile terminal that is capableof implementing a desktop window environment through an external displaydevice in a single system of the mobile terminal.

Another aspect of the present disclosure is to provide a desktopvirtualization method and apparatus of a mobile terminal that is capableof supporting large amount of information and various user experiencesfor the user.

Another aspect of the present disclosure is to provide an intelligentservice provision method and apparatus that is capable of improving userconvenience and mobile terminal usability by implementing optimalenvironment for supporting the desktop window environment in the mobileterminal.

Another aspect of the present disclosure is to provide per-displaydevice window layout provision method and apparatus of a mobile terminalthat is capable of displaying distinct screens with different windowlayouts.

Another aspect of the present disclosure is to provide a per-displaydevice window layout provision method and apparatus of a mobile terminalthat is capable of supporting different window layout screens for thedistinct display devices.

Another aspect of the present disclosure is to provide a per-displaydevice window layout provision method and apparatus of a mobile terminalthat is capable of supporting screen displays with different windowlayouts for distinct display devices according to the display deviceoutputting an execution screen of an application running on the mobileterminal.

Another aspect of the present disclosure is to provide a per-displaydevice window layout provision method and apparatus of a mobile terminalthat is capable of providing the user with various window layoutscorresponding to different display devices and supporting display ofvarious information according to the screen sizes of the displaydevices.

Another aspect of the present disclosure is to provide a per-displaydevice window layout provision method and apparatus of a mobile terminalthat is capable of improving user convenience and mobile terminalusability by implementing optimal environment for supporting screendisplays most appropriate for the respective displays especially when anapplication is running on the mobile terminal.

Another aspect of the present disclosure is to provide a method forproviding a mobile terminal user with a dual display mode using anexternal display device without extra external input device.

Another aspect of the present disclosure is to provide a method forproviding a mobile terminal user with a dual display mode using a touchscreen of the mobile terminal and an external display device.

In accordance with an aspect of the present disclosure, a functioncontrol method using a mobile terminal and an external display device isprovided. The function control method includes detecting a connection ofthe external display device, outputting a default desktop window screento the external display device, receiving a control input, controlling,when the control input is an external input for controlling a desktopregion of the external display device, a screen display of the desktopregion, and controlling, when the control input is an internal input forcontrolling a default region of the mobile terminal, a screen display ofthe default region.

In accordance with another aspect of the present disclosure, a functioncontrol method using a mobile terminal and an external display device isprovided. The function control method includes detecting a connectionbetween the mobile terminal and the external display device, executing adesktop mode, displaying a default desktop window screen for the desktopmode through the external display device, and controlling a desktopvirtualization function in a desktop window environment through theexternal display device.

In accordance with another aspect of the present disclosure, a mobileterminal is provided. The mobile terminal includes an interface unitconfigured to provide at least one of wired and wireless interface forconnection of an external display device and an external input device, atouchscreen configured to display a window screen having a layout for adefault region of the mobile terminal and a virtual input device fordesktop environmental control of a desktop region of the externaldisplay device and to receive a control input using the virtual inputdevice, and a controller configured to control desktop virtualizationfor displaying, when interoperating with the external display device,data and application generated in the mobile terminal in correspondenceto the desktop environment through the external display apparatus andprocessing operations.

In accordance with another aspect of the present disclosure, anon-transitory computer-readable storage medium is provided. Thenon-transitory computer-readable storage medium is recorded with aprogram that, when executed, causes at least one processor to perform amethod including executing, when a mobile terminal is connected to anexternal display device, a desktop mode, displaying a default desktopwindow screen for the desktop mode through the external display device,and controlling a desktop virtualization function in a desktop windowenvironment through the external display device.

In accordance with another aspect of the present disclosure, a screendisplay method using a mobile terminal and an external display device isprovided. The screen display method includes detecting a window creationevent occurring with execution of an application, determining a displaydevice for displaying a window in response to the window creation event,acquiring resource of the display device for displaying the window,determining a window layout policy of the display device correspondingto the acquired resource, creating the window on the display device in alayout according to the determined window layout policy, and displayingan execution screen of the application through the window.

In accordance with another aspect of the present disclosure, a screendisplay method using a mobile terminal and an external display device isprovided. The screen display method includes acquiring, at the mobileterminal when the external device is connected, resource of the externaldisplay device, determining a window layout policy corresponding to theacquired resource, outputting a desktop window screen determinedaccording to the window layout policy to the external display device,displaying, at the external display device, the desktop window screenaccording to the window layout policy, determining, at the mobileterminal when a user input for execution an application is detected, awindow creation region for the application, determining, when the windowcreation region is the external display device, the window layout policyfor the external display device, outputting an application executionscreen according to the determined window layout policy to the externaldisplay device, and displaying, at the external display device, theapplication execution screen in the layout according to the windowlayout policy.

In accordance with another aspect of the present disclosure, a methodfor providing a window layout per display device is provided. The methodincludes configuring window layout policies for display devices,determining, when a window is created in response to an applicationexecution request, the window layout policy to be applied to the window,and creating the window in a layout according to the determined windowlayout policy.

In accordance with another aspect of the present disclosure, a systemfor displaying a screen according to per-display device layout policiesis provided. The system includes a mobile terminal which supportsmulti-screen function in connection with an external display device andwhich outputs a window formatted according to the window layout policycorresponding to a resource of the connected external display device andan external display device which displays the window output by themobile terminal in the layout according to the window layout policy.

In accordance with another aspect of the present disclosure, a mobileterminal is provided. The mobile terminal includes an interface unitconfigured to provide at least one of wired and wireless interfaces forconnection of an external display device, a display unit configured todisplay a screen according to a preconfigured window layout policy, anda controller configured to configure window layout policies of displaydevices, to determine, when creating a window in response to anapplication execution request, the window layout policy to be applied tothe window according to the preconfigured window layout policy, tocreate the window in a layout according to the window layout policy, andto control the external display device to display the window.

In accordance with another aspect of the present disclosure, anon-transitory computer-readable storage medium is provided. Thenon-transitory computer-readable storage medium is recorded with aprogram that, when executed, causes at least one processor to perform amethod including detecting a window creation event occurring withexecution of an application, determining a display device for displayinga window in response to the window creation event, acquiring resource ofthe display device for displaying the window, determining a windowlayout policy of the display device corresponding to the acquiredresource, creating the window on the display device in a layoutaccording to the determined window layout policy, and displaying anexecution screen of the application through the window.

In accordance with another aspect of the present disclosure, a methodfor processing a user input made on a mobile terminal equipped with atouchscreen and connected to an external display device is provided. Themethod includes detecting the user input on the touchscreen, determiningwhether the user input is related to a first screen of the externaldisplay device, and performing, when the user input is related to thefirst screen, a function related to the first screen in response to theuser input and, when the user input is not related to the first screen,a function related to a second screen of the touchscreen in response tothe user input.

In accordance with still another aspect of the present disclosure, amobile terminal is provided. The mobile terminal includes an interfaceunit configured to connect to an external display device through a wiredlink, a radio communication unit configured to connect to the externaldisplay device through a wireless link, a touchscreen including a touchpanel and a display panel, and a controller configured to control theinterface unit, the radio communication unit, and the touchscreen,wherein the controller detects a user input, determines whether the userinput is related to a first screen of the external display device, andperforms, when the user input is related to the first screen, a functionrelated to the first screen in response to the user input and, when theuser input is not related to the first screen, a function related to asecond screen of the touchscreen in response to the user input.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a schematic diagram illustrating a configuration of a mobileterminal according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram illustrating connections of externaldevices to a mobile terminal according to an embodiment of the presentdisclosure;

FIG. 3 is a signal flow diagram illustrating interoperation between amobile terminal and an external display device in a display systemaccording to an embodiment of the present disclosure;

FIGS. 4 and 5 are diagrams illustrating displays of a mobile terminaland an external display device according to an embodiment of the presentdisclosure;

FIGS. 6 and 7 are diagrams illustrating displays of a mobile terminaland an external display device for explaining an input operationaccording to an embodiment of the present disclosure;

FIGS. 8, 9, 10, 11, and 12 are diagrams illustrating screen displays ofa mobile terminal and an external device for explaining a desktopvirtualization operation according to an embodiment of the presentdisclosure;

FIG. 13 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure;

FIG. 14 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure;

FIG. 15 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure;

FIG. 16 is a flowchart illustrating a per-display device window layoutprovision method of a mobile terminal according to an embodiment of thepresent disclosure;

FIG. 17 is a flowchart illustrating a per-display device window layoutmanagement method of a mobile terminal according to an embodiment of thepresent disclosure;

FIG. 18 is a signal flow diagram illustrating signal flows between amobile terminal and an external display device interoperating in amethod according to an embodiment of the present disclosure;

FIGS. 19A,19B, 19C, and 19D are diagrams illustrating screen displays ofa mobile terminal and an external display device for explaining aninteroperation there between according to an embodiment of the presentdisclosure; and

FIG. 20 is a flowchart illustrating a method for processing a user inputaccording to an embodiment of the present disclosure;

FIG. 21 is a diagram illustrating software architecture for explaining aprocedure of moving a pointer on an external display device according toan embodiment of the present disclosure;

FIG. 22 is a diagram illustrating screen displays of a mobile terminaland an external display device for explaining a procedure of moving apointer on an external display device according to an embodiment of thepresent disclosure;

FIG. 23 is a diagram illustrating software architecture for explaining aprocedure of changing a size of an image displayed on an externaldisplay device according to an embodiment of the present disclosure; and

FIG. 24 is a diagram illustrating screen displays of a mobile terminaland an external display device for explaining a procedure of changing asize of an image displayed on an external display device according to anembodiment of the present disclosure.

The same reference numerals are used to represent the same elementsthroughout the drawings.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

The present disclosure relates to a method and apparatus forimplementing desktop virtualization of a mobile terminal. According tovarious embodiments of the present disclosure, the desktopvirtualization is capable of allowing the mobile terminal interoperatingan external display device to render the data and application processedthereby to fit for the desktop window environment corresponding to theexternal display device.

In the following description, the mobile terminal may be any of all thetypes of information communication devices, multimedia devices, andequivalents thereof using an Application Process (AP), a GraphicProcessing Unit (GPU), and a Central Processing Unit (CPU). For example,the mobile terminal may be any of cellular communication terminaloperating with various communication protocols corresponding to thecommunication systems, a tablet Personal Computer (PC), a smartphone, adigital camera, a Portable Multimedia Player (PMP), a Media Player(e.g., MP3 player), a handheld e-book, a portable game console, aPersonal Digital Assistant (PDA), and the like. In addition, accordingto various embodiments of the present disclosure, the gesture-basedcontrol method may be applied to various display devices such as adigital television (TV), Digital Signage (DS), and a Large FormatDisplay (LFD), a laptop computer, and a desktop computer, and/or thelike.

Various embodiments of the present disclosure may be implemented withthe mobile terminal supporting connection to an external display deviceand the external device displaying a desktop window environment. Variousembodiments of the present disclosure may further include an inputdevice (e.g., a wired/wireless keyboard, an external mouse, and/or thelike) capable of manipulating windows presented on the external displaydevice connected to the mobile terminal.

Various embodiments of the present disclosure relate to a per-displaydevice window layout provision method and apparatus of a mobile terminalthat are capable of displaying screens having different layouts forrespective display devices. Particularly according to variousembodiments of the present disclosure, when at least one externaldisplay device is connected to the mobile terminal, the execution screenof the same application is displayed in window different layouts for themobile terminal's display unit and the external display device.

According to various embodiments of the present disclosure,differentiating among display devices based on the respective resourcesthereof (e.g., device type and screen size) is possible and configuringa window layout policy per display device is possible. According tovarious embodiments of the present disclosure, the window layout policymay be the information for determining the window layout of theexecution screen depending on the resource (particularly, screen size)of the display device.

The display method according to various embodiments of the presentdisclosure may be implemented with a mobile terminal supportingmulti-screen function and connection of external display device and anexternal display device for displaying the execution screen of anapplication running on the mobile terminal according to the windowlayout policy determined by the mobile terminal. Various embodiments ofthe present disclosure may include configuring a window layout policyper display device, determining a window layout per display device, andapplying a window layout per display device according to the windowlayout policy.

For example, according to various embodiments of the present disclosure,the display method may distinguish among the display devices accordingto the resources of the display devices and configures window layoutpolicy per display device regardless of the connection of the externaldisplay device to the mobile terminal. The window layout policy may beprovided by a provider as default or defined diversely according to theuser configuration. When a window is generated according to theexecution of an application, determining the area of the display devicefor generating a window and determining the window layout policy of thedisplay device on which the window is generated from the preconfiguredper-display device window layout policy are possible. If the windowlayout policy for the display device exists, the corresponding windowlayout policy is applied and, otherwise no window layout policy for thedisplay device exists, the default window layout policy is applied.

Hereinafter, a description is made of the configuration of the mobileterminal and control method thereof according to various embodiments ofthe present disclosure. The configuration of the mobile terminal andcontrol method thereof according to various embodiments of the presentdisclosure is not limited to the following description but may beembodied with various modifications.

FIG. 1 is a schematic diagram illustrating a configuration of a mobileterminal according to an embodiment of the present disclosure.

Referring to FIG. 1, the mobile terminal 100 includes a radiocommunication unit 110, a user input unit 120, a display unit 130, anaudio processing unit 140, a storage unit 150, an interface unit 160, acontroller 170, and a power supply 180. According to various embodimentsof the present disclosure, the mobile terminal 100 may be implementedwith or without any of the components shown in FIG. 1. If the mobileterminal 100 supports image capturing function, the mobile terminal 100may further include a camera module. If the mobile terminal 100 does notsupport the mobile communication function, a part of the radiocommunication unit 110 (e.g., cellular communication module) may beomitted.

The radio communication unit 110 may include at least one communicationmodule capable of allowing the mobile terminal 100 to communicate with aradio communication system or a network to which another device isconnected. For example, the radio communication unit 110 may include atleast one of a cellular communication module 111, a Wireless Local AreaNetwork (WLAN) module 113, a short range communication module 115, alocation positioning module 117, and a broadcast reception module 119.

The mobile communication module 111 communicates radio signals with atleast one of a base station, an external terminal, and various servers(e.g., integration server, a provider server, a content server, and/orthe like). The radio signals may include various types of data for voicetelephony, video conference, or text/multimedia messaging services. Thecellular communication module 111 connects to at least one of thevarious servers to receive an application supporting at least one ofmobile window environment and desktop window environment under thecontrol of the controller 170.

According to various embodiments of the present disclosure, the mobilewindow environment is the environment in which the function (orapplication) execution screen is rendered to be fit for the windowcorresponding to the screen size of the display unit 130 of the mobileterminal 100. The desktop window environment is the environment in whichthe function (or application) execution screen of the mobile terminal100 is rendered variously in size according to the screen size of theexternal display device for desktop environment operations.

Meanwhile, the cellular communication module 111 may connect to at leastone of the various servers to receive the window layout policy forvarious display devices under the control of the controller 170.

The WLAN module 113 is the module for connecting wireless Internet andestablishing a WLAN link with another mobile terminal. According tovarious embodiments of the present disclosure, the WLAN module 113 maybe embedded in the mobile terminal 100 or attached as an externaldevice. The WLAN module 113 may support at least one wireless internettechnology among Wi-Fi, Wireless Broadband (WiBro), WorldInteroperability for Microwave Access (WiMAX), high Speed DownlinkPacket Access (HSDPA), and the like. The WLAN module 113 may connect toat least one of the various servers to receive the applicationsupporting at least one of the mobile window environment and desktopwindow environment according to the user's selection. When a WLAN linkis established with another mobile terminal, the WLAN module 113 maytransmit or receive the application selected by the user to and from theother mobile terminal.

The WLAN module connects to at least one of the various servers toreceive the window layout policy for various display devices under thecontrol of the controller 170. If a WLAN link is established withanother mobile terminal, the WLAN module 113 may transmit or receive theper-display device window layout policies to and from the other mobileterminal according to the user's selection.

The short range communication module 115 is the module for short rangecommunication. Examples of short range communication technologiesinclude Bluetooth, Bluetooth Low Energy (BLE), Radio FrequencyIdentification (RFID), Infrared Data Association (IrDA), Ultra Wideband(UWB), ZigBee, Near Field Communication (NFC), and the like. The shortrange communication module 115 may establish a short range communicationlink with another mobile terminal to transmit or receive data to andfrom the other mobile terminal. According to various embodiments of thepresent disclosure, the short range communication module 115 may supportconnection of an external input device (e.g., Bluetooth, keyboard,mouse, and/or the like). When a short range communication link isestablished with another mobile terminal, the short range communicationmodule 115 may transmit or receive per-display device window layouts toand from the other mobile terminal.

The location positioning module 117 is responsible for positioning thelocation of the user device, and Global Positioning System (GPS) moduleis one of the representative location positioning modules. The locationpositioning module 115 collects accurate distance and time informationfrom at least three base stations and performs triangulation based onthe acquired information to acquire 3-Dimensional (3D) locationinformation with latitude, longitude, and altitude. The locationpositioning module 115 is also capable of calculating the locationinformation based on the signals from three or more satellites in realtime. The location information of the user device may be acquired usingvarious methods.

The broadcast reception module 119 receives broadcast signals (e.g., TVbroadcast signal, radio broadcast signal, and data broadcast signal)and/or information on the broadcast (e.g., broadcast channelinformation, broadcast program information, and broadcast serviceprovider information) from an external broadcast management serverthrough a broadcast channel (e.g., satellite broadcast channel, andterrestrial broadcast channel).

The input unit 120 generates an input signal for controlling theoperation of the user device in response to the user input. The inputunit 120 may include a key pad, a dome switch, a touch pad(capacitive/resistive), a jog wheel, a jog switch, and/or the like. Theinput unit 120 may be implemented with external buttons and/or virtualbutton on the touch panel.

According to various embodiments of the present disclosure, the inputunit 120 may include a plurality of keys for receiving alphanumericinformation and configuring various functions. The plurality of keys mayinclude a menu key, a screen on/off key, a power on/off key, a volumecontrol key, and/or the like. The input unit 120 generates key signalcorresponding to the user configuration and function control key inputto the controller 170. The key signal may include power on/off signal,volume control signal, and screen on/off signal. The controller 170controls the components in response to the key signal. The keys of theinput unit 120 are referred to as hard keys and the virtual keyspresented on the display unit 130 are referred to as soft keys.

The display unit 130 displays (outputs) the information processed by theuser device. For example, in the case that the user device is operatingin a telephony mode, the display unit 130 displays a telephony UserInterface (UI) or Graphic UI (GUI). In the case that the user device isoperating in a video telephony mode or a picture shooting mode, thedisplay unit 130 displays a UI or GUI displaying the picture taken bythe camera or received through the communication channel. Particularly,the display unit 130 displays the execution screens of various functions(or applications) running on the mobile terminal 100. The display unit130 presents a virtual input device (e.g., virtual touch pad) andgenerates a signal corresponding to the input made by means of thevirtual input device to the controller 170. For example, the displayunit 130 displays various execution screens of the mobile terminal 100and may present the virtual input device screen according to the user'sselection in the state that the execution screen is displayed.

The display unit 130 displays execution screens of various functions (orapplications) running on the mobile terminal 100 according to the windowlayout policy. The display unit 130 is also capable of supportingdisplay mode switching function for switching between portrait mode andlandscape mode according to the rotation direction (or orientation) ofthe user device. The operation of the display panel 131 is describedlater with reference to screens.

The display unit 130 may be implemented with any of a Liquid CrystalDisplay (LCD), a Thin Film Transistor LCD (TFT LCD), a Light EmittingDiode (LED), an Organic LED (OLED), an Active Matrix OLED (AMOLED), aflexible display, a bended display, a 3-Dimensional (3D) display, and/orthe like. The display unit 130 may be implemented as a transparentdisplay panel through which the light penetrates.

In the case that the display unit 130 and the touch panel detectingtouch gesture are layered (hereinafter, referred to as touchscreen), thedisplay unit 130 may work as an input device as well as output device.The touch panel may be configured to convert change in pressure orcapacitance at a specific region of the display unit 130 to an electricsignal. The touch panel may be configured to detect the pressure of thetouch as well as the touched position and size. If touch gesture is madeon the touch panel, the corresponding signal(s) is generated to a touchcontroller (not shown). The touch controller (not shown) processes thesignal(s) and generates corresponding data to the controller 170. Thecontroller 170 is aware of the touched region.

As described above, the display unit 130 includes the touch panel and adisplay panel. The touch panel may be placed on the display panel ofdisplay unit. In detail, the touch panel may be implemented in add-ontype in which the touch panel is placed on the display panel or inon-cell type or in-cell type in which the touch panel is embedded intodisplay panel.

The touch panel generates an analog signal (e.g., touch event) inresponse to the user's touch gesture made on the display unit 130 andconverts the analog signal to generate a digital to the controller 170.The touch event includes touch coordinates (x, y). For example, thetouch panel determines some of the coordinates within the touch area(e.g., area touched by a user's finger or a pen) as the touchcoordinates and sends the touch coordinates to the controller 170. Thetouch coordinates may be indicated by pixel. For example, if the screenresolution of the display unit 130 is 640*480, the X axis coordinate isin the range of 0 to 640 and the Y axis coordinate is in the range of 0to 480. The touch panel may generate the touch signal includingcoordinates of the touched area to the controller 170. In this case, thecontroller 170 determines some of the received coordinates as the touchcoordinates.

If the touch coordinates are received from the touch panel, thecontroller 170 determines that a touch tool (e.g., finger or pen) is incontact with the touch panel and, if no touch coordinate is received,that the touch has been released. In addition, if the coordinates arechanged, (e.g., from (x₁, y₁) to (x₂, y₂)), the controller 170determines that the touch has moved. In response to the touch movement,the controller calculates the displacement (d_(x), d_(y)) and movementspeed of the touch. The controller 170 determines the user gesture asone of touch, double touch, tap, double tap, long tap, tap & touch,drag, flick, press, pinch-in, and pinch-out, and/or the like based onthe touch coordinates, whether the touch is applied or released, whetherthe touch moves, touch displacement, touch movement speed, and/or thelike. The ‘touch’ is a user's gesture contacting a touch tool at aposition on the touchscreen, the ‘multi-touch’ is the gesture ofcontacting the touch tool at least to positions (e.g., with thumb andindex finger) on the touch screen, the ‘tap’ is the gesture ofcontacting a position on the screen with a touch tool and releasing thecontact (touch-off) without moving the touch tool, the ‘double tap’ isthe gesture of making the tap twice, the ‘long tap’ is the gesture ofmaintaining the contact for a long time as compared to the tab and thenreleasing the contact, the ‘tap and touch’ is the gesture of making atap to a certain position and then making a touch at the same positionin a predetermined time (e.g., 0.5 second), the ‘drag’ is the gesture ofcontacting a position and moving the contact on the screen in a certaindirection, the ‘flick’ is a user's gesture of snapping on the screenquickly as compared to the drag gesture, the ‘press’ is a user's gestureof contacting at a certain position on the screen and applying press,the ‘pinch-in’ is the gesture of making two contact points with twotouch tools and narrowing the distance between the two contact points,and the ‘pinch-out’ is the gesture of widening the distance between twocontact points. For example, the ‘touch’ means the state of contactingthe touch panel, and other gestures are changes in touched state. Thetouchscreen may be provided with a pressure sensor to detect thepressure at the touched position. The detected pressure information issent to the controller 170 such that the controller 170 distinguishesbetween the touch and pressure based on the pressure information.

The touch panel may be implemented as a combined touch panel including afinger touch panel for detecting a gesture made by human body and a pentouch panel for detecting pen gesture made by a pen. The finger touchpanel may be implemented as a capacitive type panel. The finger touchpanel is capable of detecting the touch gesture made by a certain object(e.g., conductive material capable of changing electrostatic capacity)as well as human body. The pen touch panel may be implemented with anelectromagnetic induction type panel. In this case, the pen touch panelgenerates a touch event in response to the gesture made by the touchstylus pen manufactured to generate magnetic field.

The display panel displays the image under the control of the controller170. The controller 170 renders the data into an image and stores theimage in a buffer. The display panel may display various imagesassociated with the operation of the mobile terminal 100 (e.g., lockscreen, home screen, application (or app) execution screen, keypad,and/or the like). If the mobile terminal powers on, the display unit 130displays the lock screen. If a touch input or key input for unlock isdetected, the controller changes the lock screen for the home screen orapplication execution screen. The home screen may include a backgroundimage and a plurality of icons arranged thereon. The icons correspond tothe respective applications. If an icon is selected by the user (e.g.,tap on an icon), the controller executes the corresponding application(e.g., browser) and displays the execution screen by means of thedisplay unit 130. The display unit 130 may display the applicationexecution screen as background and another image (e.g., keypad), on theforeground so as to be overlaid on the background. The display unit 130displays a first image at the first region and the second image at thesecond region.

The audio processing unit 140 sends the audio signal received from thecontroller 170 to the speaker (SPK) 141 and sends the audio signal suchas voice input through the microphone (MIC) 143 to the controller 170.The audio processing unit 140 is capable of processing the voice/sounddata to output an audible sound wave through the speaker 141 andprocessing the audio signal including voice to generate a digital signalto the controller 170.

The speaker 141 is capable of outputting audio received by the radiocommunication unit 110 or stored in the storage unit 150 in thetelephony mode, audio (video) recording mode, media content playbackmode, broadcast reception mode, photo capture mode, and/or the like. Thespeaker 141 is also capable of output sound effects associated with thefunction executed in the mobile terminal (e.g., inbound call reception,outbound call placing, audio and video playback, photo shooting, andexternal output).

The microphone 143 is capable of processing the input acoustic signal togenerate voice data in the telephony mode, audio (video) recording mode,speech recognition mode, photo capture mode, and/or the like. Theprocessed voice data may be processed into the signal to be transmittedto the base station by means of the cellular communication module 111 inthe telephony mode. The microphone 143 may be implemented with variousnoise cancellation algorithms to cancel the noise occurring in the audiosignal input process.

The storage unit 150 may store programs associated with the processesand controls of the controller 170 and stores input/output data (e.g.,phone number, message, multimedia contents including audio and videofiles, and applications) temporarily. The storage unit 150 may store themobile terminal's function usage frequencies (e.g., application usagefrequency, multimedia playback frequency, and phone number usagefrequency, messaging frequency, and multimedia usage frequency, weight,priority, and preference). The storage unit 150 also may store the dataassociated with the various vibration patterns and sounds correspondingto the touch gestures made on the touchscreen. According to variousembodiments of the present disclosure, the storage unit 150 may storevarious types of application for the desktop virtualization function,and the applications may be classified into the ones supporting themobile window environment and the ones supporting desktop windowenvironment. For example, certain applications performing the sameoperation may include a default mode application for outputting mobilewindow environment screen and a desktop mode application for outputtingdesktop window environment screen.

The storage unit 150 also stores the window layout policies for varioustypes of display devices to provide window layout per display device.According to various embodiments of the present disclosure, the windowlayout policies may be mapped to the display devices by resource andstored in the form of a mapping table.

The storage unit 150 stores the booting program, Operating System (OS),middleware, and a virtual controller. According to various embodimentsof the present disclosure, the virtual controller requests the OS tocreate a virtual touchscreen and controls the display unit 130 todisplay the execution screen corresponding to the virtual touchscreen.The kernel of the OS includes a U-input module. According to variousembodiments of the present disclosure, the U-input module generates thevirtual touchscreen in response to the request of the virtualcontroller, receives a touch event from the virtual controller, andsends the touch event to the virtual touchscreen. The middleware relaydata between the OS and applications or between different types ofapplications. According to various embodiments of the presentdisclosure, an Xserver is a windows OS-based middleware to receive theuser input (e.g., touch position displacement (d_(x), d_(r))) andtransfers the received user input to the application associated with theexternal display device. Thereafter, the application performs a functioncorresponding to the user input (e.g., moves the pointer in proportionto the position displacement on the external display device).

The storage unit 130 is also capable of storing embedded applicationsand third party application. The embedded applications are theapplications installed in the apparatus basically. The embeddedapplications may include browser, email, instant messenger, and thelike. The third party applications are diverse applications that may bedownloaded from the online market and installed in the terminal. Thethird party applications may be installed and uninstalled freely. If themobile terminal powers on, the booting program is loaded on the mainmemory device (e.g., RAM) of the controller 170. The booting programloads the OS and middleware of the mobile terminal onto the main memorydevice. The OS loads the applications on the main memory device toexecute.

The storage unit 150 may be implemented with a storage medium of atleast one of a flash memory type, a hard disk type, a micro type, a cardtype (e.g., a Secure Digital (SD) type and an eXtream Digital (XD) cardtype) memories, a Random Access Memory (RAM), a Dynamic RAM (DRAM), aStatic RAM (SRAM), a Read-Only Memory (ROM), a Programmable ROM (PROM),an Electrically Erasable PROM (EEPROM), a Magnetic RAM (MRAM), amagnetic disk, an optical disk type memories, and the like. The userdevice may interoperate with a web storage working as the storage unit150 on the Internet.

The interface unit 160 provides the interface for the external devicesconnectable to the user device. The interface unit 160 is capable oftransferring the data or power from the external devices to the internalcomponents of the user device and transferring the internal data to theexternal devices. For example, the interface unit 160 may be providedwith wired/wireless headset port, external charging port, wired/wirelessdata port, memory card slot, identity module slot, audio input/outputport, video input/output port, earphone jack, and/or the like. Accordingto various embodiments of the present disclosure, the interface unit 160includes a data port for connection with at least one external displaydevice through wired or wireless link. For example, the user interface160 may include High-Definition Multimedia Interface (HDMI) (includingstandard, mini, and micro HDMIs) and/or Universal Serial Bus (USB)interface. When the mobile terminal 100 and the external display deviceare connected through WLAN (Wi-Fi) link, the WLAN may be included in theinterface unit 160. When the mobile terminal 100 and the external inputdevice are connected through a Bluetooth link, the Bluetooth may beincluded in the interface unit 160.

The controller 170 controls overall operations of the mobile terminal100. For example, the controller 170 controls the operations associatedwith voice telephony, data communication, video conference, desktopvirtualization, and external output. The controller 170 may include amultimedia module (not shown) for the desktop virtualization andexternal output. According to various embodiments of the presentdisclosure, the multimedia module (not shown) may be embedded in thecontroller 170 or implemented in separation from the controller 170.

According to various embodiments of the present disclosure, thecontroller 170 controls the operations for supporting the desktopvirtualization. For example, the controller 170 controls the desktopvirtualization in which the data generated at the mobile terminal 100 inthe state of interoperation with the external display device 200 and theapplication are processed to be fit for the desktop environment of theexternal display device 200. For example, the controller 170 executes,when the mobile terminal 100 and the external display device 200 areconnected, the desktop mode and controls such that the default desktopwindow screen for the desktop mode on the external display device 200.The controller 170 controls the window screen for the mobile terminaland the window screen for the external display device with differentlayouts in control of the desktop virtualization.

When the external display device 200 is connected, the controller 170outputs the default desktop window screen to the external display device200 and monitors to detect an input of control single in the desktopmode. If a control signal for controlling the desktop area of theexternal display device is input from outside, the controller 170controls such that the desktop area is displayed on the screen. If thecontrol signal is an internal signal for controlling the default area ofthe mobile terminal, the controller 170 controls the default area screendisplay.

When the external display device 200 is connected, the controller 170may acquire the information on the screen size of the external displaydevice 200. The information on the screen size of the external displaydevice may be acquired in the form of polling of the mobile terminal 100or push of the external device 200.

The controller 170 controls displaying of the window screen with adesktop environment layout through the external display device 200 andproviding window size adjustment button(s) at the top frame of thewindow displayed on the desktop area of the external display device 200.

According to various embodiments of the present disclosure, thecontroller 170 controls the operations for supporting the functions. Forexample, the controller 170 controls such that the data and applicationof the mobile terminal 100 is displayed through the connected externaldisplay device 200 in the layout corresponding to the window layoutpolicy configured through the external display device 200. Thecontroller 170 controls the window screen displayed on the externaldisplay device 200 in separation from the window screen displayed on themobile terminal 100.

According to various embodiments of the present disclosure, thecontroller 170 controls the operation associated with the configurationof the window layout policies for the display devices. When generating awindow according to an application execution request, the controller 170may determine the layout to be applied to the window according to apredetermined window layout policy. The controller 170 generates thewindow in the layout determined according to the window layout policy tothe external display device 200.

The controller 170 may acquire the resources for the display devices andsearch for the resource per display device and window layout policymapped to the resource when determining the window. Particularly, thecontroller 170 may acquire the resource for the external display device200 in the way of polling of the mobile terminal 100 or push of theexternal display device.

According to various embodiments of the present disclosure, thecontroller 170 may include a main memory device for storing applicationprograms and OS, cache memory for storing data to be written to or readfrom the storage unit 150, a CPU, and a GPU. The OS is responsible forinterfacing between the hardware and programs and manages the computingresources such as CPU, GPU, main memory device, and auxiliary device.For example, the OS operates the mobile terminal 100, schedules tasks,and controls operations of the CPU and GPU. The OS is also responsiblefor controlling execution of application programs and managing data andfile storage. As well-known in the art, the CPU may be the main controlunit of the computer system which is responsible for data operation andcomparison and command analysis and execution. The GPU is the graphiccontrol unit for performing data operation and comparison and commandanalysis and execution in association with graphics instead of the CPU.The CPU and GPU may be integrated into a package as a single integratedcircuit composed of two or more independent cores (e.g., quad-core). TheCPU and GPU may be packaged in multi-layered manner. The integrated CPUand GPU may be referred to as Application Processor (AP).

The control operations of the controller 170 are described later indetail with reference to the accompanying drawings.

The power supply 180 supplies power of the internal or external powersource to the components under the control of the controller 170.

The various embodiments of the present disclosure may be implemented inhardware, firmware, software, or any combination thereof so as to bestored in a computer or similar device-readable storage medium. In thecase of implementing the present disclosure by hardware, the presentdisclosure can be implemented with Application Specific IntegratedCircuits (ASICs), Digital Signal Processors (DSPs), Digital SignalProcessing Devices (DSPDs), Programmable Logic Devices (PLDs), FieldProgrammable Gate Arrays (FPGAs), a processor, a controller, amicrocontroller, a microprocessor, and/or the like. In any case, thedisclosed various embodiments of the present disclosure can beimplemented on the control unit 180. In the case of implementing variousembodiments of the present disclosure by software, the proceduresfunctions disclosed in the specification can be implemented withseparate software modules. The software modules are capable ofperforming at least one function and operation described in thespecification.

The storage medium may be a non-transitory computer-readable storagemedium recording the program capable of executing a desktop mode for theexternal display device 200 connected to the mobile terminal 100,displaying default desktop window screen on the external display device200 in the desktop mode, and executing the desktop virtualizationfunction based on the desktop window environment through the externaldisplay device 200.

The storage medium may be a non-transitory computer-readable mediumrecording the program capable of detecting a window creation event inaccordance with the execution of an application, determining the displaydevice for displaying the window in response to the window creationevent, acquiring the resource for the display device, determining thewindow layout policy of the display device, generating the window on thedisplay device in the layout according to the determined window layoutpolicy, and displaying the execution screen of the application throughthe window.

According to various embodiments of the present disclosure, the mobileterminal 100 may be any of cellular communication terminal operatingwith various communication protocols corresponding to the communicationsystems, a tablet Personal Computer (PC), a smartphone, a digitalcamera, a Portable Multimedia Player (PMP), a Media Player (e.g., MP3player), a handheld e-book, a portable game console, a Personal DigitalAssistant (PDA), and the like. In addition, the gesture-based controlmethod according to one of various embodiments of the present disclosuremay be applied to various display devices such as a digital television(TV), Digital Signage (DS), and a Large Format Display (LFD).

Although it is difficult to enumerate all of the functional componentsthat can be converged in various manners according to the trend ofdigital convergence, the mobile terminal 100 may further include anacceleration sensor, a gyro sensor, a GPS module, a Near FieldCommunication (NFC) module, a vibration module, a camera, an accessory,and/or the like. The accessory may be a detachable part of the mobileterminal such as a pen to make a touch gesture on the display unit 130.A certain component of the mobile terminal 100 may be omitted orreplaced by another component according to its implementation.

A description is made of the desktop virtualization according to variousembodiments of the present disclosure with reference to FIGS. 2 to 15.

FIG. 2 is a schematic diagram illustrating connections of externaldevices to a mobile terminal according to an embodiment of the presentdisclosure.

Referring to FIG. 2, the system of the present disclosure incudes amobile terminal 100 supporting multi-screen function in connection withan external display device and the external device 200 for displayingthe screen of the mobile terminal 100 in a desktop window environment.According to various embodiments of the present disclosure, the systemmay include external input devices 300 connected to the mobile terminal100 and capable of manipulating the windows displayed on the mobileterminal 100 and/or the external display device 200. In FIG. 2, theexternal display device 200 may include a monitor, and the externalinput devices 300 may include the keyboard and mouse.

As shown in FIG. 2, the mobile terminal 100 and the external displaydevice 200 may connect through a wired interface or a wirelessinterface. For example, the mobile terminal 100 and the external displaydevice 200 may connect to each other through HDMI or Wi-Fi. The mobileterminal 100 may output various screen data to the external displaydevice 200 through the wired interface (e.g., HDMI) or the wirelessinterface (e.g., Wi-Fi)

The mobile terminal 100 and the external input devices 300 may connectthrough a wired interface or a wireless interface. For example, themobile terminal 100 and the external input devices 300 may connectthrough the wired interface such as USB or the wireless interface suchas Bluetooth. The mobile terminal 100 may receive the control signalinput by means of the external input devices 300 through the wiredinterface (e.g., USB) or the wireless interface (e.g., Bluetooth). Thecontrol signal may be the signal for controlling the window screendisplayed on the mobile terminal or the signal for controlling thewindow screen displayed on the external display device 200. According tovarious embodiments of the present disclosure, the mobile terminal 100is connected to the external display device 200 and the external inputdevices 300 as shown in FIG. 2 so as to control the window screendisplayed on the external display device 200 by means of the externalinput device 300 in the state that the desktop virtualization functionis activated in association with the external display device 200.

If a control signal is received from the external input devices 300, themobile terminal 100 controls the window screen displayed on the externaldisplay device 200 and processes operations for displaying the result onthe external display device 200 in the state of maintaining the windowscreen of the mobile terminal 100.

FIG. 3 is a signal flow diagram illustrating interoperation between amobile terminal and an external display device in a display systemaccording to an embodiment of the present disclosure, and FIGS. 4 and 5are diagrams illustrating displays of a mobile terminal and an externaldisplay device according to an embodiment of the present disclosure.

Referring to FIGS. 3, 4, and 5, at operation 301 the mobile terminal 100is connected to the display device 200.

If a connection is established between the mobile terminal 100 and thedisplay device 200 at operation 301, then at operation 303, the desktopmode is executed automatically such that the mobile terminal sends theexternal display device 200 a default desktop window screen. If thedesktop mode is executed, the mobile terminal 100 executes the desktopmode internally and transmits the default desktop window screen to theexternal display device 200 in the state of maintaining the currentdisplay of the mobile terminal 100 as shown in FIG. 4. When the externaldisplay device 200 is connected, the mobile terminal 100 acquires theinformation on the screen size of the external display device 200 anddetermines the size of the default desktop window screen correspondingto the screen size. The information on the screen size of the externaldisplay device 200 may be acquired in the form of polling or push at thetime when the external device 200 is connected.

If external display device 200 connects to the mobile terminal 100 atoperation 301 and, if the default desktop window screen is received fromthe mobile terminal 100 at operation 303, then at operation 305, theexternal display device 200 displays the default desktop window screen.The default desktop window screen may include desktop shortcut icons 250(or icon list) of certain applications capable of executing operationscorresponding to the desktop window environment through a predeterminedbackground screen 230 and the external display device 200.

If a control event for a desktop window environment task occurs, then atoperation 307, the mobile terminal 100 sends the screen corresponding tothe control event to the external display device 200. For example, ifthe user makes an input for selecting one of the desktop shortcut icons250 displayed on the external display device 200 by means of the mobileterminal 100 or the external input device 300, the mobile terminal 100determines the application represented by the selected icon 250. Themobile terminal 100 executes the determined application in the desktopmode and sends the screen created for the desktop window environment(application desktop window screen) to the external display device 200.For example, when the execution of an application represented by theicon selected on the desktop area is detected, the mobile terminal 100executes the application having the layout appropriate for the desktopenvironment.

If the external display receives the application desktop window screentransmitted by the mobile terminal 100 at operation 307, then atoperation 309, the external display device 200 displays the applicationdesktop window screen. The application desktop window screen is providedin the form of the application execution screen having the layout forthe desktop window environment different from the window screen of themobile terminal 100 as shown in FIG. 5. At this time, the screen of themobile terminal 100 may be maintained in a current state. Theapplication execution screen includes a status bar at the top of thewindow frame corresponding to the desktop window environment unlike themobile application execution screen displayed on the mobile terminal100. The status bar 550 may display the status information (applicationname, and/or the like) on the execution screen (or correspondingwindow). Particularly, the status bar 550 may include window sizeadjustment buttons 555 such as minimization, maximization, and closebuttons.

FIGS. 6 and 7 are diagrams illustrating displays of a mobile terminaland an external display device for explaining an input operationaccording to an embodiment of the present disclosure.

FIG. 6 shows screen displays in the case of using the mobile terminal100 as the input means of the external display device 200 according toan embodiment of the present disclosure, and FIG. 7 shows screendisplays in the case of using the external input device 300 connected tothe mobile terminal 100 as the input means of the external displaydevice 200 according to an embodiment of the present disclosure.

Referring to FIG. 6, the mobile terminal 100 may display a virtual inputdevice (e.g., virtual touchpad) according to the user's selection. Theuser may input a control signal using the virtual input device displayedon the mobile terminal 100 to make a manipulation such as moving thepointer 650 displayed on the external display device 200, executingapplication, moving execution window screen, enlarging/shrinking windowsize, minimizing size, maximizing size, closing window, executing adesktop shortcut icon 250, or the like.

According to various embodiments the present disclosure, the virtualinput device may provide a window size adjustment region 630 capable ofexecuting the functions of the buttons corresponding to the minimizationbutton, maximization button, and close button of the status bar 550 atthe top frame of the window displayed on the external display device200; a touchpad region 650 for detecting a gesture for moving thepointer 750 on the external display device 200, icon selection gesture,single touch gesture, and multi-touch gesture; and an indicator region610 for providing various operation status information of the mobileterminal 100 in real time. The virtual input device may also provide akeypad execution button 670 for executing the virtual touch keypad.

Referring to FIG. 7, the mobile terminal 100 may be connected with theexternal input devices 300 (e.g., keyboard and mouse) through wirelesslinks such as Bluetooth links or wired links such as USB cables. Theuser may input a control signal using the external input device 300connected to the mobile terminal 100 to perform the manipulation of suchas movement of the pointer 750 in the desktop region displayed on theexternal display device 200, application execution, executed windowmovement, window size enlargement/shrink, size minimization, sizemaximization, window close, executing a desktop shortcut icon 250, orthe like.

FIGS. 8, 9, 10, 11, and 12 are diagrams illustrating screen displays ofa mobile terminal and an external device for explaining a desktopvirtualization operation according to an embodiment of the presentdisclosure.

Referring to FIG. 8, a screen display in the state that an applicationis running in the desktop window environment of the external displaydevice 200 under the control of the mobile terminal 100 is illustrated.As shown in FIG. 8, the application execution window provides a statusbar 550 having a window size adjustment button 555 at the tope side ofthe window frame. FIG. 8 is directed to the case in which the externalinput device 300 is connected to the mobile terminal 100 formanipulating the window displayed on the external display device 200such that the pointer 750 moves on the desktop region according to theinput mode of the external input device 300.

When an application execution command is input by selecting an icon inthe desktop region of the external display device 200, the mobileterminal 100 executes the application having the layout for the desktopenvironment and processes the corresponding screen output. If theapplication desktop window screen is received from the mobile terminal100, the external display device 200 displays the application desktopwindow screen. The application desktop window screen is the applicationexecution screen formed in the layout fit for the desktop environmentunlike the window screen of the mobile terminal 100 as shown in FIG. 8.At this time, the screen of the mobile terminal 100 may be maintained ina current state.

Referring to FIG. 9, a screen display in the state that pluralapplications are executed in the desktop window environment of theexternal display device 200 is illustrated. Particularly, when theplural application execution windows may be provided with a status bar550 having the window size adjustment buttons 555 at the top sides oftheir frames respectively. The status bar 550 of each window may providethe status information of the corresponding application. FIG. 9 isdirected to the case in which the window displayed on the externaldisplay device 200 is adjusted by means of the external input device 300connected to the mobile terminal 100 and the pointer 750 is provided onin the desktop region according to the input mode of the external inputdevice 300.

Referring to FIG. 10, a screen display in the case of extending theexecution window of the application according to the user's manipulationin the state that the application is executed in the desktop windowenvironment through the external display device 200 is illustrated. Theapplication execution window may be provided with a status bar 550. Thestatus bar 550 may further include window size adjustment buttons 555.As shown in FIG. 10, the user may move the pointer 750 to the right mostposition of the window frame by means of the external input device 300.When the pointer 750 is placed at the edge, the pointer 750 is changedin shape as an indicator to indicate that the window size can beextended or shrunk. The user may make a control signal for rightdirection movement by manipulating the external input device 300. Thenthe window is extended as shown in FIG. 10. For example, if the windowis extended in the state that a gallery list is displayed in the windowaccording to the execution of the gallery application, the window screenis provided in such a way of including the region presenting detailedinformation (enlarged photo) on the photos of the gallery list.

Referring to FIG. 11, screen displays in the case of moving anapplication execution screen according to the user's manipulation in thestate that the application is executed in the desktop window through theexternal display device 200 under the control of the mobile terminal 100are illustrated. As shown in FIG. 11, the user may move the pointer 750at the top side of the window frame (e.g., the status bar 550) using theexternal input device 300. When the pointer 750 is placed at a positionon the status bar 550, the pointer 750 may be changed in shape toindicate that the window may be moved. In this state, the user may makea control input for movement in a direction (e.g., rightward) in thedisplay region of the external display apparatus 200 by manipulating theexternal input device 300. Then the window moves in response to the userinput as shown in FIG. 11.

Referring to FIG. 12, screen displays in the state that a specificapplication is executed in the desktop window environment through theexternal display device 200 under the control of the mobile terminal 100are illustrated. As shown in FIG. 12, the application execution windowmay be provided with the status bar 550 having the window sizeadjustment button 555. Particularly, FIG. 12 shows the case in which theapplication executed in the region of the mobile terminal 100 in thestate of FIG. 8. As shown in FIG. 12, if an application execution eventsuch as incoming call reception in the state that the mobile terminal100 is connected to the external display device 200, the screen of theexternal display device 200 displays the window screen (e.g., incomingcall reception status screen) on the mobile terminal 100 according tothe execution event at the mobile terminal in the state of maintain thecurrent state. As shown in FIG. 12, as the application is executed inthe mobile terminal 100, the pointer 750 may disappear on the currentscreen of the external display device 200. This is to indicateintuitively that the current manipulation is performed on the mobileterminals.

If an application execution command input by the selection of the iconon the desktop region of the external display device 200 is detected,the mobile terminal 100 executes an application having the desktopenvironment-friendly layout and processes the screen display. If theapplication desktop window screen is received from the mobile terminal100, the external device 200 displays the application desktop windowscreen having the desktop environment-friendly layout. If theapplication is executed in the region of the mobile terminal, the mobileenvironment-friendly application screen is displayed as shown in FIG.12.

Although FIGS. 8 to 12 are directed to the case in which the externalinput device 300 is connected to the mobile terminal 100, the desktopwindow environment of the external display device 200 may be manipulatedthrough the virtual input device provided on the mobile terminal 100 asdescribed above with the omission of the external input device 300.

FIG. 13 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure.

Referring to FIG. 13, at operation 1301, the controller 170 determines acurrent operation mode (e.g., standby mode and application executionmode).

At operation 1303, the controller 170 detects a connection of theexternal device 200. For example, the controller 170 may detect theconnection of the external display device 200 through a wired (HDMI)link or a wireless (Wi-Fi) link. If the external display device 200 isconnected, the controller 170 executes the desktop mode for desktopvirtualization and transmits a default desktop window screen to theexternal display device 200. If the external display device 200 isconnected, the controller 170 may determine the type of the connectionbetween the mobile terminal 100 and the external display device 200.

When the external display device 200 is connected to the mobile terminal100 at operation 1303, the controller 170 proceeds to operation 1305 atwhich the controller 170 acquires the information on the screen size ofthe external display device 200. The information on the screen size ofthe external display device 200 is acquired in the way of polling orpush at the time when the mobile terminal and the external displaydevice 200 connect to each other.

At operation 1307, the controller 170 determines the size of the defaultdesktop window screen based on the information on the screen size of theexternal display device 200.

At operation 1309, the controller 170 outputs the default desktop windowscreen corresponding to the determined screen size to the externaldisplay device 200. The controller 170 may process the output of thedefault desktop window screen through wired (HDMI) or wireless (Wi-Fi)link depending on the connection type of the external display device200.

At operation 1311, the controller 170 detects a control input.

If a control input is detected in the state that the default desktopwindow screen for desktop mode operation is displayed on the externaldisplay device 200 at operation 1311, then the controller 170 proceedsto operation 1313 at which the controller 170 determines whether thecontrol input is an internal control input for controlling the defaultregion of the mobile terminal 100 or an external control input forcontrolling the desktop region of the external display device 200.

If the controller 170 determines that the control input is not theexternal control input (e.g., if the control input is the internalcontrol input), at operation 1313, then the controller 170 proceeds tooperation 1315 at which the controller 170 controls such that theexecution screen is displayed in the mobile terminal-friendly layout inthe default region of the mobile terminal while the current screen ofthe external display device 200 is maintained. For example, thecontroller 170 controls displaying the window screen of the defaultregion of the mobile terminal 100.

Conversely, if the controller 170 determines that the control input isthe external control input at operation 1313, then the controller 170proceeds to operation 1317 at which the controller 170 controls suchthat the desktop window screen is displayed on the desktop region of theexternal display device 200 in the desktop environment-friendly layout.

FIG. 14 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure.

Referring to FIG. 14, a flowchart is illustrated in a case in which noexternal input device is connected to the mobile terminal 100. However,the mobile terminal 100 provides a virtual input device for manipulatingthe data region of the external display device.

Referring to FIG. 14, at operation 1401, the controller 170 determinesthe execution of the desktop mode at operation 1401. For example, thecontroller 170 determines whether the mobile terminal 100 is operatingin the desktop mode.

At operation 1403, the controller 170 determines whether a request forvirtual input device execution is detected.

If a virtual input device execution request is detected in the desktopmode at operation 1403, the controller 170 proceeds to operation 1405 atwhich the controller 170 executes the virtual input device application.For example, the controller 170 may receive the input requesting for theexecution of the virtual input device through manipulation of the mobileterminal (e.g., menu manipulation and shortcut key input) in the statein which the default desktop window screen is displayed on the externaldisplay device 200. Then the controller 170 controls the mobile terminal100 such that the virtual input device application is executed in thedefault region of the mobile terminal 100 in the state of maintain thedefault desktop window screen of the external display device 200.

At operation 1407, the controller 170 displays the virtual input deviceon the mobile terminal 100 and the pointer on the external displaydevice 200. For example, the controller 170 outputs the virtual inputdevice in match with the layout of the default region of the mobileterminal 100 with the execution of the virtual input device applicationand controls to present the pointer 750 in the desktop region of theexternal display device 200.

At operation 1409, the controller 170 controls the desktop regionaccording to the user input made through the virtual input device in thestate in which the virtual input device is displayed. For example, thecontroller 170 may controls the operations as described with referenceto FIGS. 8, 9, 10, 11, and 12. For example, the controller 170 maycontrol the tasks of the desktop environment such as pointer movement inthe desktop region, application execution, executed window screenmovement, window size extension/shrink, size minimization, sizemaximization, window close, and/or the like.

FIG. 15 is a flowchart illustrating a desktop virtualization method of amobile terminal according to an embodiment of the present disclosure.

Referring to FIG. 15, the operations of processing the input from theexternal input device 300 in the state in which the external device 300is connected to the mobile terminal 100 is illustrated.

At operation 1501, the controller 170 determines the execution of thedesktop mode.

At operation 1503, the controller 170 receives a control input madethrough the external input device 300.

If a control input is received from the external input device 300 in thedesktop mode at operation 1503, then the controller 170 proceeds tooperation 1505 at which the controller 170 maintains the current stateof the screen of the default region of the mobile terminal 100.

Thereafter, at operation 1507, the controller 170 determines the controlevent of the external input device 300. For example, the controller 170is capable of determining whether the control input is made through thekeyboard or the mouse and which task is to be executed according to thecontrol input.

At operation 1509, the controller 170 controls output of the desktopregion according to the identified control input. For example, thecontroller 170 may control the movement of the pointer in the desktopregion of the external display device 200, application execution,executed window screen movement, window size enlargement/shrink, sizeminimization, size maximization, window close, or the like whilemaintaining the current state of the screen of the mobile terminal 100.

Descriptions are made of the various embodiments related to theper-display device window layout provision function hereinafter withreference to FIGS. 16, 17, 18, and 19A, 19B, 19C, and 19D.

FIG. 16 is a flowchart illustrating a per-display device window layoutprovision method of a mobile terminal according to an embodiment of thepresent disclosure.

Referring to FIG. 16, at operation 1601, the controller 170 detects aninput requesting to execute an application.

If the controller 170 detects a request for an application executioninput by the user at operation 1601, then the controller 170 proceeds tooperation 1603 at which the controller 170 acquires the resource of thedisplay device on which the application is executed. For example, if theapplication execution request is detected, the controller 170 identifiesthe display device on which the application execution screen is to bedisplayed and acquires the resource of the display device. The resourceof the display device is the resource of the display devices currentlyconnected to the mobile terminal 100 (e.g., display unit 130 of themobile terminal 100 and external display device connected to the mobileterminal 100) and acquired from the storage unit 150 or the externaldisplay device.

If the controller 170 acquires the resource of the display device onwhich the application execution screen is displayed at operation 1603,then the controller 170 proceeds to operation 1605 at which thecontroller 170 determines a window layout policy configured for theacquired resource. For example, the controller 170 may retrieve apredetermined window layout policy corresponding to the resource of thedisplay device from the mapping table of the storage unit 150.

At operation 1607, the controller 170 decorates (e.g., configures) thewindow layout of the application according to the determined windowlayout policy. For example, the controller 170 configures the layout ofthe window presenting the application execution screen according to thewindow layout policy.

At operation 1609, the controller 170 generates the window decorated(e.g., configured) according to the window layout on the display deviceand displays the application execution screen.

As described with reference to FIG. 16, when the user executes anapplication, the controller 170 acquires the resource of at least onedisplay device currently connected to the mobile terminal 100 andselects one of the window layout policies configured based on theresource of the display device. The controller 170 decorates (e.g.,configures) the window layout of the executed application according tothe determined window layout policy.

FIG. 17 is a flowchart illustrating a per-display device window layoutmanagement method of a mobile terminal according to an embodiment of thepresent disclosure.

Referring to FIG. 17, at operation 1701, the controller 170 detects thewindow generation request of the user.

At operation 1703, the controller 170 acquires the resource of thecurrently connected display device. For example, if the user'sapplication execution request is detected, the controller 170 maydetermine that the application execution request corresponds to thewindow generation event for generating a new window. The controller 170determines the display device for which the window is generatedaccording to the application execution (e.g., display unit 130 of themobile terminal 100 and external display device connected to the mobileterminal 100) and acquires the resource of the determined displaydevice. The resource of the display device may include the resource ofat least one of the display unit 130 of the mobile terminal and theexternal display device connected to the mobile terminal 100.

At operation 1705, the controller 170 searches for the window layoutpolicy for the resource of the display device.

At operation 1707, the controller 170 determines whether the windowlayout policy is retrieved. For example, the controller 170 scans theresources of the display devices in correspondence to the resourceacquired from the mapping table stored in the storage unit 150 andretrieves the window layout policy mapped to the display device of theacquired resource.

If the controller 170 retrieves a window layout policy for the displaydevice at operation 1707, then the controller 170 proceeds to operation1709 at which the controller 170 determines that the found window layoutpolicy corresponds to the window layout policy of the window to bedisplay on the display device.

At operation 1711, the controller 170 applies the determined windowlayout policy to the window to be displayed on the display device.

At operation 1713, the controller 170 displays the window to which thewindow layout policy is applied on the display device. For example, thecontroller 170 generates the window according to the applicable windowlayout policy.

Conversely, if the controller 170 does not retrieve a window layoutpolicy for the display device at operation 1707, then the controller 170proceeds to operation 1715 at which the controller 170 determines thatthe default window layout policy corresponds to the window layout policyof the window to be displayed on the display device.

At operation 1717, the controller 170 applies the default window layoutpolicy to the window to be displayed on the display device. Thereafter,the controller 170 proceeds to operation 1713 at which the controller170 displays the window to which the window layout policy is applied onthe display device.

Referring to FIG. 17, the controller 170 may determine whether anywindow layout policy of the display device on which the window is to bedisplayed when an application execution window is generated in responseto the user request exists. If the window layout policy of the displaydevice is retrieved, the controller 170 applies the retrieved windowlayout policy as the window layout policy of the display device.Otherwise, if no window layout policy is retrieved, the controller 170applies the default window layout policy as the window layout policy ofthe display device. The controller 170 generates the window to which thewindow layout policy is applied and provides the application executionscreen according to the layout of the window.

FIG. 18 is a signal flow diagram illustrating signal flows between amobile terminal and an external display device interoperating in amethod according to an embodiment of the present disclosure. FIGS. 19A,19B, 19C, and 19D are diagrams illustrating screen displays of a mobileterminal and an external display device for explaining an interoperationthere between.

Referring to FIGS. 18, 19A, 19B, 19C, and 19D, at operation 1801, themobile terminal 100 is connected to the external display device 200. Forexample, the mobile terminal 100 may detects the connection of theexternal display device 200 through a wired (e.g., HDMI) link or awireless (e.g., Wi-Fi) link in a certain operation mode (e.g., standbymode and application execution mode).

If the external display device 200 is connected to the mobile terminal100 at operation 1801, then at operation 1803, the mobile terminal 100acquires the resource of the external display device 200. For example,if the external display device 200 is connected, the mobile terminal 100acquires the resource such as screen size of the external device 200.The resource of the external display device 200 may be provided in sucha way that the mobile terminal 200 request the connected externaldisplay device 200 to transmit the resource and the external displaydevice 200 provides the mobile terminal 100 with its resource inresponse to the resource transmission request. The external displaydevice 200 may transmit the resource thereof to the mobile terminalautomatically when the external display device 200 is connected to themobile terminal 100. According to various embodiments of the presentdisclosure, the resource of the external display device is acquired inthe way of polling or pushing the resource at the time when the mobileterminal 100 is connected to the external display device 200, theresource including the screen size of the external display device 200.The mobile terminal 100 may buffer or store the acquired resource in thestorage unit 150 of the mobile terminal 100.

At this time, the host providing the resource may be a slave which doesnot perform any control function but provides screen data under thecontrol of a master. The master may be the terminal 100 responsible forthe control (particularly the controller 170), and the slave theexternal display device 200 and the display unit 130 of the mobileterminal 100 that display the screen data provided by the mobileterminal 100.

At operation 1805, the mobile terminal 100 determines the window layoutpolicy corresponding to the resource acquired from the external displaydevice 200. For example, the mobile terminal 100 may determine thewindow layout policy corresponding to the acquired resource byreferencing the predetermined mapping table of the per-display devicewindow layout policies.

At operation 1807, the mobile terminal 100 outputs the desktop windowscreen (e.g., background screen) to the external display device 200according to the determined window layout policy. When displaying thedesktop window screen, the mobile terminal 100 transfers the desktopwindow screen generated internally according to the window layout policyto the external display device 200 in the state of the current displayof the mobile terminal 100 (e.g., current operation mode screendisplay).

After being connected to the mobile terminal 100, if the desktop windowscreen is received from the mobile terminal 100, then at operation 1809,the external display device 200 displays the desktop window screen. Asillustrated in FIG. 19A, the desktop window screen may include thedesktop shortcut icons 250 representing the pre-registered applicationsso as to execute operation with the window corresponding to the resourceof the predetermined background screen 230 and the external displaydevice 200.

In the state in which the external display device 200 is connected tothe mobile terminal 100, at operation 1811, the mobile terminal mayreceive a user input for executing an application. For example, the usermay select an icon representing a specific application on the screen(e.g., a current screen of the mobile terminal 100 in FIG. 19A)displayed currently on the display device (e.g., display unit 130) ofthe mobile terminal 100 by manipulating the mobile terminal 100. Theuser also may manipulate the mobile terminal 100 to select an icon of aspecific application (e.g., shortcut icon 250 for desktop) on thecurrently displayed screen of the external display device 200 (e.g.,desktop window screen currently displayed on the external display device200 in FIG. 19A).

At operation 1813, the mobile terminal 100 determines the windowcreation region for the application in response to the applicationexecution request. For example, if an application execution request isdetected, the mobile terminal 100 may determine the display device onwhich the application execution screen is to be displayed. For example,if the application execution request is detected on the currentlydisplay screen of the mobile terminal 100, the mobile terminal 100 maydetermine that the window creation region is the display device of themobile terminal 100 (e.g., the display unit 130), and if the applicationexecution request is detected on the currently displayed screen of theexternal display device 200, the mobile terminal may determine that thewindow creation region is the external display device 200. Although theapplication execution request is detected on the screen of the mobileterminal 100 in the state in which at least one external display device200 is connected, the external display device 200 may be determined asthe window creation region. The determination that the window creationregion corresponds to the external display device 200 may depend on theuser configuration.

If the window creation region is determined, then at operation 1815, themobile terminal 100 determines the window layout policy corresponding tothe display device as the window creation region. For example, themobile terminal 100 may reference the mapping table of the per-displaydevice window layout policies to select the window layout policycorresponding to the resource of the display device as the windowcreation region (e.g., display unit 130 of the mobile terminal 100 andexternal display device 200). FIG. 18 is directed to the case in whichthe window creation region for the executed application is the externaldisplay device 200.

At operation 1817, the mobile terminal 100 outputs the applicationexecution screen corresponding to the determined window layout policy tothe external display device 200. For example, the mobile terminal 100configures the window layout of the executed application according tothe determined window layout policy and outputs the applicationexecution screen corresponding to the window layout to which the windowlayout policy has been applied. For example, the mobile terminal 100determines the application represented by the icon 250 selected by theuser among the icons 250 displayed on the external display device 200.The mobile terminal 100 configures the application execution window withthe layout according to the determined window layout policy andtransfers the application execution screen to the external displaydevice 200. For example, the mobile terminal 100 may execute theapplication with the layout corresponding to the resource of the displaydevice (e.g., the external display device 200) in determining theexternal display device 200 as the application window creation regionand process the screen display.

If the application execution screen is received from the mobile terminal100 at operation 1817, then at operation 1819, the external displaydevice 200 displays the application execution screen. The applicationexecution screen is provided in the form of a window screen having thelayout fit for the resource of the external display device 200 differentfrom the window screen displayed on the display device (e.g., displayunit 130) of the mobile terminal 100 as shown in FIG. 19B. For example,the application execution screen may be implemented in the form ofdesktop window layout according to the resource of the external displaydevice 200. Referring to FIG. 19B, the desktop window may have the sizelarger than that displayed on the mobile terminal 100 along withdetailed information. The further supplementary information may beprovided at the top side of the frame of the window. The status bar 550may present the status information (e.g., application name) on thecorresponding window (e.g., execution screen). Particularly, the statusbar 550 may include the window size adjustment buttons 555 such asexecuted window minimization and maximization, and window close buttons.At this time, the mobile terminal 100 may maintain the screen in thecurrent state thereof. Further, the desktop window screen may includethe desktop shortcut icons 250 representing the pre-registeredapplications so as to execute operation with the window corresponding tothe resource of the predetermined background screen 230 and the externaldisplay device 200.

FIG. 19C is a diagram illustrating screen displays of the mobileterminal and the external display device with distinct window layoutsaccording to an embodiment of the present disclosure.

As shown in FIG. 19C, the system is configured with the mobile terminal100 supporting multi-screen function in connection with the externaldisplay device 200 displaying the screen output by the mobile terminal100 in the window layout determined according to the per-display devicewindow layout policy.

In FIG. 19C, the mobile terminal 100 and the external display device 200are connected to each other through a wired interface (e.g., HDMI) or awireless interface (e.g., Wi-Fi). The mobile terminal 100 transmitsvarious screen data to the external display device 200 through the wiredinterface (e.g., HDMI) or the wireless interface (e.g., Wi-Fi).

In FIG. 19C, a certain application is executed with the execution screenwhich is displayed by the external display device 200 according to theexternal display device-specific window layout policy and by theinternal display device of the mobile terminal 100 (e.g., display unit130) according to the mobile terminal-specific window layout policyunder the control of the mobile terminal 100

Referring to FIG. 19C, the mobile terminal in the state of beingconnected to the external display device 200 may provide the externaldisplay device 200 with the application execution screen to which thewindow layout policy corresponding to the resource of the externaldisplay device is applied and provide the display unit 130 of the mobileterminal 100 with the application execution screen to which the windowlayout policy corresponding to the resource of the display unit 130 ofthe mobile terminal (e.g., default window layout policy). For example,the application execution screen may be provided in different windowlayouts determined according to the display device-specific windowlayout policies.

If an application execution request is detected on the region of theexternal display device 200, the mobile terminal executes theapplication in the window format based on the window layout policymatching the resource of the external display device 200 and processesthe output of the corresponding window. If the application executionrequest is detected on the region of the display unit 130 of the mobileterminal 100, the mobile terminal 100 executes the application in thewindow format based on the window layout policy matching the resource ofthe display unit 130 and processes the out of the corresponding window.In the case of generating the window according to the applicationexecution at the external display device 200, the mobile terminal 100formats the application execution screen in match with the resource ofthe external display device 200. In the case of generating the windowaccording to the application execution at the mobile terminal 100, themobile terminal 100 formats the application execution screen in matchwith the resource of the display unit 130.

FIG. 19D is a diagram illustrating screen displays of the mobileterminal and the external display device with distinct window layoutsaccording to an embodiment of the present disclosure.

FIG. 19D shows the screen displays in the state in which the mobileterminal 100 and the external display device 200 are connected through awired link (e.g., HDMI) or a wireless link (e.g., Wi-Fi) and theexecution screen of a certain application is displayed across the regionof the mobile terminal 100 and the region of the external display device200. For example, in the state in which the application is executed withthe execution window displayed on the region of the external displaydevice 200, the application execution window is moved to the region ofthe mobile terminal 100 in response to the user's window movemanipulation such that the application execution window appearspartially on the region of the mobile terminal 100.

As shown in FIG. 19D, a part of the execution screen of an applicationmay be displayed on the region of the mobile terminal 100 as formattedin the layout according to the window layout policy of the display unit130 of the mobile terminal, while the other part of the execution screenis displayed on the region of the external display device 200 asformatted in the layout according to the window layout policy of theexternal display device 200. For example, the execution window of thesame application may be display in distinct layouts depending on thedisplay device-specific window layout. In a case of a photo galleryapplication, the application execution window for the external displaydevice 200 may be displayed in the layout having a list of photos anddetailed description of a selected photo (e.g., enlarged photo), and theapplication execution window for the mobile terminal 100 may bedisplayed in the layout having only the list of the photos.

Although not shown in FIG. 19D, the user may manipulate such that theapplication execution window is moved from the region of one displaydevice to the region of the other display device completely. In thiscase, the application execution window may be changed in layout from thewindow layout policy of the old display device to the window layoutpolicy of the new display device. For example, if the applicationexecution window is moved from the region of the external display device200 to the region of the mobile terminal 100 completely according to theuser's manipulation, the application execution window may be formattedin the window layout according to the mobile terminal-specific windowlayout policy. In contrast, if the application execution window is movedfrom the region of the mobile terminal 100 to the region of the externaldisplay device 200 completely, the application execution window may beformatted in the window layout according to the external displaydevice-specific window layout policy.

Descriptions are made of the various embodiments related to the displayfunction of the external display device in response to the user input indetail hereinafter with reference to FIGS. 20 to 24.

FIG. 20 is a flowchart illustrating a method for processing a user inputaccording to an embodiment of the present disclosure.

Referring to FIG. 20, at operation 2010, the controller 170 determinesthat the mobile terminal 100 is operating in the single display mode.

At operation 2015, the controller 170 determines whether any externaldisplay device is connected to the mobile terminal 100. For example, theexternal display device may be connected to the mobile terminal 100through the radio communication unit or the interface unit 160.

If the controller 170 determines that an external display device isconnected to the mobile terminal 100, then the controller 170 proceedsto operation 2020 at which the controller 170 controls the mobileterminal 100 to operate in the dual display mode. At operation 2020, ifthe external display device is connected to the mobile terminal 100, thecontroller 170 sends the external display device a message requestingfor screen size information (e.g., resolution information). In responseto the request, the external device sends the mobile terminal 100 aresponse message including the screen size information. The display unit130 displays a first home screen including an icon of a virtualcontroller under the control of the controller 170. The external displaydevice displays a second home screen under the control of the controller170. At this time, the resolution of the second home screen isdetermined based on the screen size information received from theexternal display device. For example, the second home screen isformatted to match the screen size of the external display device. Ifthe user makes a tap gesture to the icon of the virtual controller onthe first home screen, the display unit 130 generates an input signalcorresponding to the tap gesture to the controller 170. The controller170 receives the input signal and executes the virtual controller inresponse to the input signal. The virtual controller requests the kernelto generate a virtual touchscreen. The kernel, particularly the U-inputmodule, generates a virtual touchscreen for emulating the display unit130 as the input device of the external display device. Meanwhile, thedisplay unit 130 may display the execution screen of the virtualcontroller (e.g., emulation screen), under the control of the controller170. At this time, the emulation screen includes a touch region. Theemulation screen also may include at least one soft key (e.g., leftmouse button, right mouse button, minimization button, maximizationbutton, and close button). The emulation screen may be displayed in theform of an overlaid window. In addition, the emulation screen may bedisplayed on the second screen region while another screen is displayedin the first screen region. The emulation screen is closed in responseto a user's request. If the user selects the corresponding icon on thefirst home screen, the emulation screen appears again. The virtualcontroller may be executed automatically when the external displaydevice is connected to the mobile terminal 100. The auto-executionfunction may be turned on/off by the user. In the case in which theemulation screen is displayed, the controller 170 controls the externaldisplay device to display a pointer on the other screen (e.g., thesecond home screen) in the form of overlay. If the emulation screenfunction is terminated, the pointer disappears.

At operation 2030, the controller 170 monitors to detect a user input.

At operation 2035, the controller 170 determines whether a user input isdetected on the display unit 130. The user input may be any of touch,multi-touch, tap, double tap, long tap, drag, flick, press, pinch-in,pinch-out, and/or the like.

If the controller determines that the user input is not detected on thedisplay unit 130 at operation 2035, then the controller 170 proceeds tooperation 2030.

If the controller 170 determines that the user input is detected on thedisplay unit 130 at operation 2035, then the controller 170 proceeds tooperation 2040 at which the controller 170 determines whether the userinput is related to the external display device. For example, if noemulation screen is displayed, the controller 170 determines that theuser input is not related to the external display device. Otherwise, ifthe emulation screen is displayed, the controller 170 determines thatthe user input is related to the external display device. If the userinput is detected on the first screen region in the state in which theemulation screen is displayed on the first screen region and anotherscreen is displayed on the second screen region, the controller 170determines that the user input is related to the external displaydevice. If the user input is detected on the second screen region, thecontroller 170 determines that the user input is not related to theexternal display device.

If the controller 170 determines that the user input is related to theexternal display device at operation 2040, then the controller proceedsto operation 2050 at which the controller 170 performs a functionrelated to the screen of the external display device in response to theuser input. For example, if a drag gesture is detected on the touchregion of the emulation screen, the controller 170 moves the pointer inthe drag direction. For example, the external display device shows thepointer in motion under the control of the controller 170. If a tapgesture is detected on the touch region of the emulation screen or aleft mouse button is clicked in the state in which in pointer is placedon touch region, the controller 170 executes the application representedby the icon on which the pointer is placed and controls the externaldisplay device to display the execution screen. If the user input is the“tap & touch followed by drag,” the controller 170 moves the pointeralong with the execution window on which the pointer is placed. Asdescribed above with reference to operation 2050, the display unit 130is used as the input device for the external display device.

If the controller 170 determines that the user input is not related tothe screen of the external display device (e.g., if the user input isrelated to the screen of the display unit 130) at operation 2040, thenthe controller 170 proceeds to operation 2060 at which the controller170 performs a function related to the screen of the display unit 130 inresponse to the user input. For example, if the user input is a drag,the controller 170 changes the screen according to the drag direction.For example, the first page of the first home screen may be replaced bythe second page. If the user input is a tap, the controller 170 executesthe application represented by the icon tapped and controls the displayunit 130 to display the execution screen.

At operation 2070, the controller 170 determines whether a dual displaymode termination request is input. For example, when the connectionbetween the external display device and the mobile terminal 100 isreleased, the dual display mode is terminated. Otherwise, the controller170 returns to operation 2030.

FIG. 21 is a diagram illustrating software architecture for explaining aprocedure of moving a pointer on an external display device according toan embodiment of the present disclosure. FIG. 22 is a diagramillustrating screen displays of a mobile terminal and an externaldisplay device for explaining a procedure of moving a pointer on anexternal display device according to an embodiment of the presentdisclosure.

Referring to FIG. 21, the controller 170 includes a virtual controller310, a U-input module 320, a virtual touchscreen 330, a virtualmulti-touchscreen 340, an Xserver 350, and an application 360.

At operation 1, the virtual controller 310 is executed.

At operation 2, the virtual controller 310 controls the display unit 130to display an emulation screen including an emulation region 410, a leftmouse button 420, a right mouse button 430, a minimization button 440, amaximization button 450, and an end button 460.

At operation 3, the virtual controller 310 transmits a request for avirtual touch screen creation to the U-input module 320.

At operation 4, the U-input module 320 creates a virtual touch screenand a virtual multi-touch touch screen.

At operation 5, the touch coordinates (x_(n), y_(n)) are transferred tothe Xserver 350.

At operation 6, the Xserver 350 forwards the touch coordinates (x_(n),y_(n)) to the virtual controller 310.

At operation 7, the virtual controller 310 determines whether the(x_(n), y_(n)) is the coordinates of the emulation region 410.

If the touch coordinates (x_(n), y_(n)) are the coordinates of theemulation region 410, then at operation 8, the virtual controller 310calculates the touch displacement (d_(x)(x_(n)−x_(n-1)),d_(y)(y_(n)−y_(n-1))). (x_(n-1), y_(n-1)) are the previously detectedtouch coordinates.

At operation 9, the virtual controller 310 sends the positiondisplacement (d_(x), d_(y)) to the U-input module 320.

At operation 10, the U-input module 320 sends the position displacement(d_(x), d_(y)) to the virtual touchscreen 330.

At operation 11, the virtual touchscreen 330 sends the positiondisplacement (d_(x), d_(y)) to the Xserver 350.

At operation 12, the Xserver 350 sends the position displacement (d_(x),d_(y)) to the application 360 associated with the external displaydevice. The application moves the pointer 470 from the first position480 to the second position 490 in response to the position displacement(d_(x), d_(y)) as shown in FIG. 22.

FIG. 23 is a diagram illustrating software architecture for explaining aprocedure of changing a size of an image displayed on an externaldisplay device according to an embodiment of the present disclosure.FIG. 24 is a diagram illustrating screen displays of a mobile terminaland an external display device for explaining a procedure of changing asize of an image displayed on an external display device according to anembodiment of the present disclosure.

Referring to FIG. 23, the controller 170 includes a virtual controller510, a U-input module 520, a virtual touch screen 530, a virtualmulti-touch screen 540, an Xserver 550, and an application 560.

Operations 1 to 4 of FIG. 23 are identical with those of FIG. 21.

At operation 5, the first touch coordinates (x1, y1) and the secondtouch coordinates (x2, xy) are sent to the Xserver 550.

At operation 6, the Xserver 550 sends the first and second touchcoordinates (x1, y1) and (x2, y2) to the virtual controller 510.

At operation 7, the virtual controller 510 determines whether the firstand second touch coordinates (x1, y1) and (x2, y2) are the coordinatesin the emulation region 610.

If the first and second touch coordinates (x1, y1) and (x2, y2) are thecoordinates in the emulation region 610, then at operation 8, thevirtual controller 510 determines the size (A) of the emulation region610 and the size (B) of the execution window 620 (see FIG. 6) andrespectively converts the first and second touch coordinates (x1, y1)and (x2, y2) to the third and fourth touch coordinates (x3, y3) and (x4,y4).

At operation 9, the virtual controller 510 sends the third and fourthtouch coordinates (x3, y3) and (x4, y4) to the U-input module 520.

At operation 10, the U-input module 520 sends the virtualmulti-touchscreen 540 the third and fourth touch coordinates (x3, y3)and (x4, y4).

At operation 11, the virtual multi-touchscreen 540 sends the Xserver 550the third and fourth touch coordinates (x3, y3) and (x4, y4).

At operation 12, the Xserver 550 sends the application related to theexternal display device the third and fourth touch coordinates (x3, y3)and (x4, y4). The application 560 changes (e.g., enlarges) the size ofthe execution window 620 based on the third and fourth touch coordinates(x3, y3) and (x4, y4).

As described above, according to various embodiments of the presentdisclosure, the desktop virtualization method and apparatus of themobile terminal provide a desktop virtualization function capable ofallowing the mobile terminal to perform desktop environment operation ininteroperation with the an external display device. Accordingly, thedesktop virtualization method and apparatus according to variousembodiments of the present disclosure are capable of providing the userwith the desktop window environment for presenting more information andmeeting various user requirements. According to various embodiments ofthe present disclosure, the desktop virtualization method and apparatusare capable resolving the inconvenience caused by the size-constrainedscreen of the mobile device and allowing the user to use the mobileterminal in the desktop environment on the large screen of the connectedexternal display device.

According to various embodiments of the present disclosure, the desktopvirtualization method and apparatus are capable of providing anoptimized desktop window environment to the mobile terminal user so asto improve the user convenience and usability and competitiveness of themobile terminal. According to various embodiments of the presentdisclosure, the desktop virtualization method and apparatus areapplicable to all the types of the mobile terminals and equivalentdevices supporting multi-screen function.

According to various embodiments of the present disclosure, theper-display device window layout provision method and apparatus of themobile terminal are capable of providing the user with the screendisplay in various window layouts fit for various display devices.According to various embodiments of the present disclosure, the methodand apparatus are capable of providing the display device-specificscreen layouts so as to improve the user convenience.

According to various embodiments of the present disclosure, theper-display device window layout provision method and apparatus of thepresent disclosure are capable of configuring distinct window layoutpolicies corresponding to at least two display devices such that anapplication is executed the execution screen of an application is formedin the display device-specific window layout according to the windowlayout policy matching the display device. According to variousembodiments of the present disclosure, the method and apparatus arecapable resolving the inconvenience caused by the size-constrainedscreen of the mobile device and allowing the user to use the mobileterminal in the desktop environment on the large screen of the connectedexternal display device.

According to various embodiments of the present disclosure, the methodand apparatus are capable of providing optimal environment fordisplaying the screen in the format matching the display device so as toimprove the user convenience and usability and competitiveness of themobile terminal. According to various embodiments of the presentdisclosure, the method and apparatus are applicable to all the types ofthe mobile terminals and equivalent devices supporting multi-screenfunction.

According to various embodiments of the present disclosure, the userinput processing method and apparatus of the mobile terminalinteroperating with an external display device are capable of providingthe user with a dual display mode environment without extra externalinput device. According to various embodiments of the presentdisclosure, the method and apparatus are capable of providing the userwith the dual screen mode environment using a touchscreen of the mobileterminal and an external display device.

The above-described various embodiments of the present disclosure can beimplemented in the form of computer-executable program commands andstored in a non-transitory computer-readable storage medium. Thenon-transitory computer readable storage medium may store the programcommands, data files, and data structures in individual or combinedforms. The program commands recorded in the storage medium may bedesigned and implemented for various embodiments of the presentdisclosure or used by those skilled in the computer software field.

The non-transitory computer-readable storage medium includes magneticmedia such as a floppy disk and a magnetic tape, optical media includinga Compact Disc (CD) ROM and a Digital Video Disc (DVD) ROM, amagneto-optical media such as a floptical disk, and the hardware devicedesigned for storing and executing program commands such as ROM, RAM,flash memory, and the like. The programs commands include the languagecode executable by computers using the interpreter as well as themachine language codes created by a compiler. The aforementionedhardware device can be implemented with one or more software modules forexecuting the operations of the various embodiments of the presentdisclosure.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. A function control method using a mobile terminaland an external display device, the method comprising: detecting aconnection of the external display device; outputting a default desktopwindow screen to the external display device; receiving a control input;controlling, when the control input is an external input for controllinga desktop region of the external display device, a screen display of thedesktop region; and controlling, when the control input is an internalinput for controlling a default region of the mobile terminal, a screendisplay of the default region.